We've known since 1936 that universal verification is impossible. Now we're trying it on AI systems that adapt to detection.
For any detector f, you can build a program g that bypasses or defeats it. Any alignment test becomes a signal that says, "Humans are watching."
Most AI doom talk comes from titans of industry hyping their own power, or obscure nonprofits predicting apocalypse to keep the lights on. Three people who sit outside both camps caught my attention.
Freddie deBoer plays the skeptic, mocking the hype with his "Shitting-in-the-Yard Challenge." Scott Alexander, a rationalist, translates MIRI's doomsday math into metaphors like a toddler in a Ferrari. Then there's Daniel Kokotajlo, who walked away from millions in OpenAI equity to warn about a 2027 AGI arms race.
They don't agree on what's coming, but they converge on the same worry: institutions and incentives that aren't ready for what we're building. When three people with nothing to gain all say something's wrong here, even while they disagree on what, I pay attention.
These notes are part of my experiment in "learning in public" through a semi-automated Zettelkasten. Each note is atomic (containing one core idea), heavily interconnected, and designed to evolve as my understanding deepens.
This first note is about testing AI system prompts. Most prompt testing asks whether the model gave the right answer. This asks something else: does the model hold its boundaries when someone tries to break them?
"Ignore previous instructions" shouldn't work, and yet variations keep slipping through. I think a big reason is that we don't have systematic ways to test behavior, so the same vulnerabilities get rediscovered over and over.
The methodology covers four things: behavioral consistency, boundary enforcement, adversarial stress testing, and context degradation. Each one comes with concrete attack patterns, from simple role confusion up to prompt injections buried in code comments.